Machine learning is the science of inducing computing systems to act without being explicitly programmed. Classical machine learning includes various clustering and classification techniques, including K-means clustering, linear and logistic regressions, stochastic gradient decent, association rule learning, and the like. Deep learning is a newer frontier in machine learning. Deep learning is a class of machine learning algorithms that uses multiple layers of nonlinear processing units for feature extraction and transformation. Deep learning algorithms may be unsupervised (e.g., pattern analysis) or supervised (e.g., classification). The deep learning algorithm may be implemented using layers of an artificial neural network (ANN) (referred to herein as a “neural network”).
In general, a neural network is a collection of nodes (i.e., the “neurons”) that are connected (e.g., in a graph). A node in a neural network may compute a sum of weighted inputs and may add an optional bias to the sum. The output of the node is a function of the final sum (referred to as an “activation function”). Example activation functions include the sigmoid function, the hyperbolic tangent (tan h) function, the Rectified Linear Unit (ReLU) function, and the identity function. Neural network models are often organized into layers of nodes, which define a specific topology, and corresponding weights and biases. The weights and biases are referred to as network parameters.
In general, a neural network includes an input layer and an output layer and may optionally include one or more hidden layers between the input and output layers. A neural network used in deep learning applications typically includes many hidden layers, which gives rise to the term deep neural network (DNN). The layers of a neural network may be densely connected (e.g., each node in a layer is fully connected to all nodes in a previous layer) or sparsely connected (e.g., each node in a layer is connected to only a portion of the nodes in a previous layer). A convolutional neural network (CNN) is a type of DNN that includes one or more sparsely connected layers, referred to as convolutional layers. A CNN is well-suited for processing image or video data. Other types of DNNs include recurrent neural network (RNNs), which are well-suited for processing speech and text data.